PVM–based Training of Large Neural Architectures

In this paper a methodology for parallelizing neural network training algorithms is described, based on the parallel evaluation of the error function and gradient using the Parallel Virtual Machine (PVM). PVM is an integrated set of software tools and libraries that emulates a general–purpose, flexible, heterogeneous concurrent computing framework on interconnected computers of varied architectures. The proposed methodology has large granularity and low synchronization, and has been implemented and tested. Our results indicate that the relatively easy setup of the PVM (using existing workstations), and parallelization of the training algorithms results in considerable speedups especially when large network architectures and training vectors are used.